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RECOGNIZING FILM AND VIDEO OBJECTS 
OCCURRING IN PARALLEL IN SINGLE TELEVISION SIGNAL FIELDS 

BACKGROUND OF THE INVENTION 
Field Of The Invention 

[0001] The present invention relates to the field of detecting 
motion picture film sources in film material. 

5 

Description Of The Related Art 

[0002] In U.S. Patent 5,734,735, a method and system is 
described that analyzes a series of video images. The types of 
production media used to produce these video images are detected. 

10 Each of the series of video images is segmented into a series of 
cells in order to retain spatial information. The spatial 
information is used to detect the type of production media. No 
technique is disclosed to detect types of production for different 
scenes within one image, coming from different sources and being 

15 mixed to form the single image. 

[0003] U.S. Patent 6,014,182 also relates to methods for 
detecting motion picture film sources. Such a detection might be 
useful in several environments, e.g., line doublers, television 
standard converters, television slow motion processing and video 

20 compression. For instance, a 60 Hz NTSC television signal has a 24 
frame/second motion picture film as its source. In such a scheme, a 
3-2 pull-down ratio is used, i.e., three video fields come from one 
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film frame while the next two video fields come from the next film 
frame, etc. For example, calling subsequent video fields A, B, C, 
D, E, a 3-2 pull-down ratio would look like AAABBCCCDDEEE . Other 
sources have a 2-2 pull down ratio or relate to video camera, as is 
5 known to persons skilled in the art. Thus, comparing successive 
fields yields information about the motion picture source used. 
[0004] U.S. Patent 5,365,280 proposes using different motion 
vectors for different fields and generating a picture signal 
processing mode control signal that can be used by a television 
10 receiver as an indication that the fields relate either to movie- 
film or non-movie- film. 

[0005] Motion estimation algorithms can be found in M. Tekalp, 
"Digital Video Processing", Prentice Hall, ISBN 0-13-190075-7. An 
overview of object -based motion estimation methods is given by 
15 Paolo Vicari, "Representation and regular izat ion of motion fields 
with region-based models", thesis for the Politecnico di Milano, 
no. 598034. 

[0006] So far, the prior has concentrated on detecting motion 
picture sources of either films having fields originating from a 

20 single motion picture source or films having subsequent fields 
originating from two or more different motion picture sources. 
However, an increasing number of films comprise mixtures of images 
within fields that originate from different motion picture sources. 
None of the prior art methods discussed above, are able to detect 

25 the picture repetition mode of individual images within fields of a 
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film. For instance, in applications in picture rate conversion, 
however, an indication of the origin of the individual images 
within the fields needs to be known. More particularly, it is 
necessary to know whether the video originates from film material 
5 to optimally perform de- interlacing and film judder removal. 

SUMMARY OF THE INVENTION 
[0007] Therefore, it is an objective of the present invention to 
provide an apparatus and a method for detecting the picture 

10 repetition mode of individual objects within fields. In this 

context, an "object" may be a portion of an individual image in a 
field. An "object" is defined as an image portion that can be 
described with a single motion model. Such an "object" need not 
necessarily comprise one "physical" object, like a picture of one 

15 person. An object may well relate to more than one physical object, 
e.g., a person sitting on a bike where the movement of the person 
and the bike, essentially, can be described with the same motion 
model. On the other hand, one can safely assume that objects 
identified in this way belong to one single image originating from 

20 one single film source. 

[0008] To obtain the objective of the present invention, it 
provides a method to detect a picture repetition mode of film 
material comprising a series of consecutive fields, the method 
comprising the following steps: 
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[0009] Establishing a motion parameter pattern for the film 
material ; 

[0010] Comparing the pattern with a number of predetermined 
motion parameter patterns; 
5 [0011] Determining the picture repetition mode using the result 
of the preceding step; 

characterized in that, the method includes the following steps: 
[0012] Identifying a plurality of different objects within the 
consecutive fields, an object being defined as an image portion of 
10 the consecutive fields that can be described with a single motion 
model ; 

[0013] Carrying out the following steps: 

[0014] Establishing a motion parameter pattern for each one of 
the objects within the consecutive fields; 
15 [0015] Comparing the motion parameter pattern with a number of 
predetermined motion parameter patterns; and 

[0016] Determining the picture repetition mode for each one of 

the objects using the result of the preceding step. 

[0017] Thus, in accordance with the present invention, prior to 

20 detecting a film mode, the fields of the television signal are 
separated into different objects by means of a segmentation 
technique. Any known technique to do so might, be used for that 
purpose. Then, the film mode of each individual object is detected. 
Any known film mode detection technique might be used for that 

25 purpose. 



N : \UserPublic\GO\SC05GOA0 . GOR 4 



I > 

PHNL 010005 

[0018] Preferably, a motion parameter estimation technique is 
used as well . 

[0019] In particular, the invention uses the technique of motion 
parameter estimation to identify different image portions (objects) 
5 originating from different sources because of mixing. 

[0020] The invention also relates to an arrangement for 
detecting a picture repetition mode of film material comprising a 
series of consecutive fields, the arrangement comprising processing 
means and a memory, the processing means being arranged to carry 
10 out the following steps: 

[0021] Establishing a motion parameter pattern for the film 
material ; 

[0022] Comparing the pattern with a number of predetermined 
motion parameter patterns stored in the memory; 
15 [0023] Determining the picture repetition mode using the result 
of the preceding step; 

characterized in that, the processing means are arranged to carry 
out the following steps: 

[0024] Identifying a plurality of different objects within the 
2 0 consecutive fields, an object being defined as an image portion of 
the consecutive fields that can be described with a single motion 
model ; 

[0025] Carrying out the following steps: 

[0026] Establishing a motion parameter pattern for each one of 
25 the objects within the consecutive fields; 
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[0027] Comparing the motion parameter pattern with a number of 
predetermined motion parameter patterns stored in the memory; and 
[0028] Determining the picture repetition mode for each one of 
the objects using the result of the preceding step. 
5 [0029] Such an arrangement may, advantageously, be implemented 
on a chip, and such a chip may be arranged in a 
television. 

[0030] The invention also relates to a computer program product 
to be loaded by a computer arrangement, comprising instructions to 

10 detect a picture repetition mode of film material comprising a 

series of consecutive fields, the arrangement comprising processing 
means and a memory, the computer program product, after being 
loaded, providing the processing means with the capability to carry 
out the following steps: 

15 [0031] Establishing a motion parameter pattern for the film 
material ; 

[0032] Comparing the pattern with a number of predetermined 
motion parameter patterns stored in the memory; 

[0033] Determining the picture repetition mode using the result 
20 of the preceding step; 

characterized in that, the processing means are arranged to carry 
out the following steps: 

[0034] Identifying a plurality of different objects within the 
consecutive fields, an object being defined as an image portion of 
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the consecutive fields that can be described with a single motion 
model ; 

[0035] Carrying out the following steps: 

[0036] Establishing a motion parameter pattern for each one of 
5 the objects within the consecutive fields; 

[0037] Comparing the motion parameter pattern with a number of 
predetermined motion parameter patterns stored in the memory; and 
[0038] Determining the picture repetition mode for each one of 
the objects using the result of the preceding step. 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0039] The invention will now be explained with reference to the 
accompanying drawings that are only intended to illustrate the 
present invention and not to limit its scope, in which: 
15 [0040] Figure 1 shows a block diagram of a multiple parameter 
estimator and segmentation arrangement; 

[0041] Figures 2A, 2B, 2C, 2D show television screen photographs 
illustrating a process of selecting points of interest on which 
parameter estimators optimise their parameters; and 
20 [0042] Figures 3A, 3B, 3C, 3D show television screen photographs 
illustrating a process of segmentation. 
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DESCRIPTION OF THE PREFERRED EMBODIMENT 

Introduction 

[0043] Hereinafter, a method to detect a film mode of individual 
objects in a scene is proposed. To that end, first of all, a method 
5 is described to identify individual objects in a scene. Individual 
objects are identified by motion estimation, i.e., those portions 
of a scene that can be described with a same motion model are 
identified as belonging to a same object in the scene. Motion 
estimators are known as such from the prior art, e.g., from Refs. 

10 [1], [3], [4], [5], and [6]. Of these references, Ref. [1] 

describes a motion estimator enabling identification of objects in 
a scene without the need to apply image segmentation. 
[0044] For the present invention, a motion estimator is 
preferred that is designed to be suitable for picture rate 

15 conversion, with a computational complexity suitable for consumer 
electronics application, i.e., comparable to Refs. [5] and [6]. 
[0045] The most striking characteristic of the object motion 
estimator described earlier in Ref. [1], is that no effort is put 
in segmenting the image into objects prior to estimation of the 

20 model parameters, like in other prior art object motion estimators. 
Basically, a relatively small number of interesting image parts is 
selected, and a number of parallel motion model parameter 
estimators is trying to optimize their parameters on this data set. 
As soon as one of the estimators is more successful than another in 

25 a certain number of interesting image parts, it is focused on those 
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parts, whereas the remaining estimators focus on the other parts. 
In short, individual estimators try to conquer image parts from one 
another, dividing the total image into "objects". This prior art 
object motion estimator allows a real-time object-based motion 
5 estimation and can advantageously be used in the film detection 
technique of the present invention. 

[0046] Fundamentally, such an object-based motion estimator that 
wastes no effort in expensive segmentation of the image, should be 
able to compete in operations count with a block-based motion 

10 estimator, as one should expect less objects than blocks in 

realistic images. It is only in the assignment of image parts to 
objects that an effort is required comparable to the evaluation of 
candidate vectors on block basis. If the number of objects does not 
exceed the number of candidate vectors too much, the overhead of an 

15 object-based motion estimator should be negligible. It is assumed 
here that the motion per object can be described with fairly simple 
parametric models. 

[0047] In the following subsections, we shall describe a 
preferred motion model used, an estimation of motion model 
20 parameters, a preferred cost function used, a segmentation process 
and a film-mode detection for individual objects within a scene. 

Motion Model 

[0048] To keep complexity low, the motion of each object o is 
25 described by a simple first -order linear model that can only 
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describe translation and scaling. More complex parametric motion 
models are known to persons skilled in the art, e.g., models 
including rotation, and can indeed be applied in combination with 
the proposed algorithm, but will be disregarded here, as we shall 
introduce a refinement that makes such complex models obsolete. 
[0049] The model used is: 



< s x (o,n) + xd x {o,ri)^ 
y s y {o,n) + yd y {o,n) J 



(1) 



10 using D 0 (x,n) for the displacement vector of object o at location 



x = 



in the image with index n. It is observed that x is 



associated with pixel locations. 



Parameter Estimation 

15 [0050] Given a motion model, next its parameters need to be 
optimized for a given object in the image. As stationary image 
parts occur in almost every sequence, we assume the presence of an 
'object o, o > 0 1 , for which motion is described by 0, the zero 
vector. Clearly, no estimation effort is required to make this 

2 0 available. The parameter vectors of additional objects o, o > 0, 
are estimated separately, in parallel, by their respective 
parameter estimators (PE m , m = 1,2,..., M) , as shown in Figure 1. 
[0051] Figure 1 shows a block diagram of an arrangement with a 
plurality of parameter estimators PE m (n) connected in parallel to 

25 the output of a data reduction unit DRU. The data reduction unit 
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DRU is arranged to select a set of interesting image pixels that 
are to be used in the calculations. Inputs to the data reduction 
unit DRU are the image at time n and the image at time n-1. Each of 
the outputs of the parameter estimators PE m (n) is connected to a 
5 segmentation unit SU. 

[0052] The output of the segmentation unit SU is fed back to the 
parameter estimators PE m (n) since, preferably, they together 
perform a recursive operation as will be explained below. The end 
result of the segmentation process is formed by groups of pixels of 

10 a scene, each group of pixels belonging to a different object and 
having assigned to it a different motion vector. These output data 
are supplied to a processing unit PU that is arranged to detect the 
type of film source per object and to perform predetermined tasks 
on the different objects such as picture rate conversion. The 

15 processing unit PU is connected to memory M storing predetermined 
motion parameter patterns used to detect the type of film source as 
will be explained below. The memory M may be of any known type, 
i.e., RAM, ROM, EE PROM, hard disc, etc. The output of the 
processing unit PU, for instance, controls a television screen. 

20 [0053] It is observed that the data reduction unit DRU, the 
parameter estimators PE m (n) , the segmentation unit SU and the 
processing unit PU are shown as separate blocks. These blocks may 
be implemented as separate intelligent units having distinct 
processors and memories. However, as is evident to persons skilled 

25 in the art, these units may be integrated into a single unit such 
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as a general purpose microprocessor comprising a processor and 
suitable memory loaded with suitable software. Such a 
microprocessor is not shown but known from any computer handbook. 
Alternatively, the arrangement shown in Figure 1 may be implemented 
5 as a hard-wired logic unit, as known to persons skilled in the art. 
Preferably, the entire arrangement shown in Figure 1 is 
encapsulated as a single chip in a single package. Such a single 
chip package can be easily included in a television apparatus. 
[0054] Each parameter estimator PE m (n) updates a previously 
10 estimated parameter vector, after which, the best parameter 

candidate vector, according to a cost function, is selected as the 
result parameter vector for that object. 

[0055] Considering the four parameter model of equation (1) , the 
parameters of object o, o > 0, are regarded as a parameter vector 
15 P»: 

r s x (o,n)^ 

(2) 



PM = 



s y {o,n) 
d x (o,n) 
K d y (o,n) 

and we define as our task to select a parameter vector P 0 (n) from a 
20 number of candidate parameter vectors C 0 {n) as the one that has the 
minimal value of a cost function, to which we shall return later 
on. 

[0056] Preferably, the candidates are generated much similar to 
the strategy exploited in Refs. [5] and [6], i.e., take a 
25 prediction vector, add at least one update vector, and select the 
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best candidate parameter vector according to an error criterion. 
Candidate parameter set CS Q (n) contains three candidates C 0 (n) 
according to: 



CS 0 in) = jc 0 (n) C 0 (n) = P 0 (n-l) + mU 0 (n), U 0 (n) e US, (n), m = -1,0,1 } 



(3) 



with update parameter U 0 (n) selected from update parameter set 
US c (n) : 



10 US 0 {n) = 

(i=l,2,4, 8,16) 
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(4) 



The Cost Function 

15 [0057] Given the motion model and some candidate parameter sets, 
we need to select the best candidate, according to a cost function, 
as the result for a given object. The cost function can be a sum of 
absolute differences between motion- compensated pixels from 
neighboring images, with vectors generated with the (candidate) 

20 motion model. However, we need to know the area to which the motion 
model is to be assigned. The two issues, segmentation and motion 
estimation, are inter-dependent. In order to correctly estimate the 
motion in one object, the object should be known and vice versa. 
[0058] As a first step in the motion estimation process, we 

25 define a set with pixel blocks of interest. These form the set 
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SI(n) of "interesting" image parts that will be used as a basis for 
optimization of all parametric models. 

[0059] Now, the focus of the individual parameter estimators has 
to be on different objects. To this end, each parameter estimator 
5 PE m (n) will calculate its cost function on the same set of 

interesting locations defined in set SI, giving different locations 
a different weight factor, W Q (X) . Here, X is associated with a 
position of a block of pixels. The proposed algorithm is 
straightforward : 

10 [0060] The pixel values are multiplied with a first weight 

factor larger than 1, e.g., 8, in case the pixel in SI (n) belonged 
to the same object, i.e., the same parameter estimator, according 
to the previous image segmentation step. 

[0061] The pixel values are multiplied with a second weight 
15 factor smaller than 1, e.g., 0.1, in case the segmentation assigned 
the position to another parameter estimator and this estimator 
achieved low match errors. 

[0062] Figures 2A-2D give an example of a selection of pixel 
blocks of interest in an image with a single moving object, i.e., a 

20 bicyclist, and a moving background. This selection is carried out 
by the Data Reduction Unit DRU. Thus, the Data Reduction Unit 
renders a set of most interesting pixel elements (SI) , resulting in 
a rather cheap (few calculations) and an effective parameter 
estimation. Figures 2A-2D show screen photographs illustrating a 

25 process of selecting points of interest on which the parameter 
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estimators optimize their parameters. The temporal difference 
image, between two successive pictures, is not actually calculated, 
but it serves to understand why the high match errors of the vector 
6, i.e., the total set with points of interest, are at the 
5 positions shown in Figure 2C. In Figure 2D, it is shown how, in 

this example, the focus of two parameter estimators is divided over 
the points of interest. That is, Figure 2D shows that there are two 
different motion models detected. The two sub- sets are shown in 
different brightnesses, i.e., one in black and the other one in 



the bicyclist is object o = 2. There are two parameter estimators 
that are both optimized on the same set containing the blocks of 
interest, but as soon as one estimator is selected in the 
15 segmentation to be best in an area, the pixel block of interest in 
that area is emphasized in the cost function. After a while, this 
converges to the situation illustrated, where one estimator focuses 
on the gray blocks and the other on the white pixel blocks in 
SI(n) . 

20 [0064] More formally, the cost function is calculated according 



10 



gray. 



[0063] 



The moving background of the image is object o = 1, and 



to: 



£(C 0 , n) = X W o & n) -F s (x- C 0 (x, n\ n - 1)| 



(5) 
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10 



where F s (x,n) is the luminance value of a pixel at position x in a 
sub- sampled image with index n, and C 0 (x,n) is the vector resulting 
from candidate model C Q (n) at position x . 

[0065] The sub- sampling effectively reduces the required memory 
bandwidth. Images are sub- sampled with a factor of four 
horizontally and a factor of two vertically on a field base, 
generating a sub-sampled image F s (n) from each original field F(n). 
In order to achieve pixel accuracy on the original pixel grid of F, 
interpolation is required on the sub- sampling grid. 



Recursive Segmentation 

[0066] The segmentation is the most critical step in the 
algorithm. Its task is to assign one motion model to each group of 
pixels. For each block, a block match error, s Q {X,ri) , corresponding 
15 to each of the estimated parameter vectors, P Q , can be calculated 
according to: 

e(X,n)= £ \F s (x^(l-a)D 0 (x,nln)-F s (x-aD 0 (x y n%n-l\ (6) 

xgB(X) 

20 [0067] The temporal instance where this segmentation is valid is 
defined by a. 

[0068] We adopted a recursive segmentation method that closely 
resembles the strategy of a 3-D RS block matcher, e.g., as 
disclosed in Ref. [5], i.e., use spatial and temporal predictions 
25 of the best PE m (n) , and penalize choosing a PE m (n) that does not 
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occur in the spatio-temporal neighborhood. Formally, the 
segmentation mask M(X,n) assigns the object o with the lowest local 
modified cost function e 0 (X,n) to the block B(X) , where: 

5 =e 0 +P(X,n) (7) 



10 



while P(X,n) is a penalty chosen according to the following rule: 

' P s ,(M(X + 8,n) = o) 
P(X,n) = lp t ,(M(X-5,n-\) = o) (8) 
P u , (otherwise) 

and 



S = 



,/,y = o,±i O) 



15 [0069] Similar to what has been suggested for the 3-D RS block 
matcher (Ref. [5]), P u is the largest penalty, P t just a small one, 
while there is no reason why P s could not just be zero. A fairly 
obvious simplification is to fix 8 to the direction opposite to 
the scanning direction, and to alternate the scanning from field to 

20 field. Figure 3A-3D give an example of segmentation according to 
the object-based motion estimation method, with the original 
luminance image. Figures 3A-3D show photographs taken from a 
television screen and illustrating the process of segmentation. 
Figure 3A shows the original image while Figures 3B-3D show 

25 consecutive segmentation results. Clearly, the first image, in 
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Figure 3A, has a poor, almost random, segmentation. However, the 
focusing of the individual estimators to their area in the 
segmentation rapidly converges to a useful segmentation: Figure 3D 
shows that two different objects can be distinguished, one relating 
5 to the bicyclist and one relating to the back ground. 

Prior Art Film-Mode Recognition 

[0070] Apart from the calculation of motion vectors for every 
object in the picture, applications in picture rate conversion 

10 require an indication of the origin of the picture sequence. More 
particularly, it is necessary to know whether the video originates 
from film material to optimally perform de-interlacing and film 
judder removal. Moreover, it is necessary to distinguish between 
2-2 pull down image material, 2-3 pull down material, and video 

15 from a video camera. 

[0071] As discussed earlier here, in prior art methods, this 
detection concerns a global decision, i.e., discrimination between 
video camera and the various film formats is done only for entire 
images . 

20 [0072] As an adaptation of Ref. [8], for the object-based motion 
estimator, a reliable movie detector can be realized analyzing the 
motion described by the parameter estimator only that covers the 
largest area of the image, obviously disregarding the zero-vector 
1 estimator 1 . 
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[0073] Let us define max(n) as the largest component of 
parameter vector P Q (n) (rather than taking the largest component of 
the parameter vector, it is equally well possible to use the 
average, absolute, or the summed absolute value of either or both 
5 of the parameter components), i.e.: 

max(n) = max{s x (o,n) , s y (o,n) , d x (o,n) , d y (o,n)} (10) 

[0074] We now assemble the recent history set RH(n) as: 



which, with adaptive thresholding, is converted into a binary movie 
15 detection set MD(n), that, for 2-2 pull-down, will gives something 



10 



RH(n) = (max(n) , max(n-l), max(n-2), max(n-3), max(n-4) , 
max(n-5) , max(n-6) } 



(11) 



like : 



MD(n) = {0,1,0,1,0,1,0}, 



(12) 



20 for 2-3 pull --down, something like: 



MD(n) = {0,1,0,0,1,0,1}, 



(13) 



and for video, something like: 



25 
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MD(n) = {1,1,1,1,1,1,1}. (14) 

[0075] Comparing the actual set with a limited number of known 
patterns stored in memory M, yields information on movie type and 
5 phase. In case of scene cuts, the detector yields the output 

unreliable, which indicates that motion compensation can better be 
switched off. 

Film-Mode Recognition According To The Invention 
10 [0076] The invention concerns a method to detect the film mode 
of individual objects in a scene. More and more, images from 
different sources are mixed during production. We, therefore, 
propose to adapt the object-based motion estimator such that it, 
along with the motion parameter estimation of the objects in the 
15 scene, decides upon their origin. 

[0077] To this end, we analyze the motion described by all 
individual parameter estimators . 

[0078] Let us define max Q (n) as the largest component of 
parameter vector P Q (n) (rather than taking the largest component of 
20 the parameter vector, it is equally well possible to use the 

average, absolute, or the summed absolute value of either or both 
of the parameter components), i.e.: 

max Q (n) = max{s x (o, n) , s y (o,n) , d x (o,n), d y (o,n)}. (15) 

25 
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[0079] We now assemble the recent history sets RH Q (n) as: 

RH Q (n) = {max 0 (n), max 0 (n-l), max Q (n-2), max Q (n-3), 

max 0 (n-4) / max Q (n-5), max Q (n-6) (16) 

5 

which, with adaptive thresholding, are converted into binary movie 
detection sets MD Q (n), that, for a 2-2 pull-down object, will give 
something like: 

10 MD Q (n) = {0,1,0,1,0,1,0}, (17) 
for 2-3 pull-down, something like: 

MD Q (n) = {0,1,0,0,1,0,1}, (18) 

15 

and for video, something like: 

MD 0 (n)= {1,1,1,1,1,1,1}. (19) 

20 [0080] Comparing the actual set with a limited number of known 
patterns stored in memory M, yields information on movie type and 
phase for every individual object. In case of scene cuts, the 
detector yields the output unreliable, which indicates that motion 
compensation can better be switched off for all objects. 

25 
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